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ABSTRACT 



This final report describes the outcomes of Project MAPS 
(Monitoring Authentic Problem Solving) , a project that developed multimedia 
computer-managed assessment environments to overcome the barriers educators 
face in using problem-solving performance assessment (PSPA) with students 
with disabilities. MAPS software was developed for 6 alternate performance 
assessments at each grade: grades 2, 3, 4, 5, and 6, for a total of 30 
multimedia performance assessments. The multimedia presentation includes CD, 
options for reviewing selected parts of the CD, options for selecting parts 
of the text view, and options for selecting parts of the text for the 
computer to read/reread aloud. Students represent their responses on the 
computer by moving items from a sticker gallery or by typing in answers. The 
interactive scoring system automatically scores the student's responses. A 
study in which 40 students with disabilities and 40 comparable, same-age 
students completed performance assessments with and without the MAPS 
multimedia system, found the MAPS system could provide support for students 
to show what they know on complex performance assessments. Also, results 
showed that students could be trained in the use of the software in large 
group sessions lasting no more than 45 minutes. Appendices include sample 
screens from the MAPS multimedia system. (CR) 
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PURPOSE AND CONTEXT 



The national education reform movement seeks to enhance student performance on real-world 
problem-solving (PS) tasks, rather than on isolated skills. This shift has prompted leaders to reject 
well established assessment models because of decontextualized testing formats. Instead, leaders 
have called for a new assessment model, known as problem-solving performance assessment 
(PSP A), which requires students to solve authentic problems by applying multiple skills and 
strategies to real-life situations. This PSPA movement has gained tremendous momentum; over 40 
states have incorporated PSP As into high-stakes accountability programs. 

For students with disabilities (SWD), who typically have poorly developed strategies for applying 
skills and who increasingly are expected to participate in state accountability programs, this PSPA 
movement, along with its goal of redirecting instruction to facilitate skill application and 
generalization to naturalistic contexts, is critical. Teachers, therefore, need feasible classroom-based 
PSP As with which to routinely track improvement in PS capacity and to use that assessment 
information to plan better programs. 

Unfortunately, barriers currently prevent teachers from having PSP As that feasibly and accurately 



index PS capacity among SWD and facilitate better PS instructional planning for SWD. For 
example, available PSPAs rely on text-based presentation and response formats, spuriously 
lowering estimates of PS capacity for SWD (who have reading and writing deficits). PSPAs are 
time consuming for teachers to score and interpret, limiting their usefulness in schools. Teachers 
have difficulty connecting PSPA information with productive teaching methods for improving PS 
performance. And, PSPAs do not provide parallel forms for tracking PS improvement over time or 
for providing SWD with routine opportunities for practice. 

The purpose of "Monitoring Authentic Problem Solving: MAPS for Enhancing Outcomes for 
Students with Disabilities," therefore, was to develop multimedia computer-managed assessment 
environments to overcome these barriers. These systems comprise multimedia presentations of 
problem situations and interactive dialogues whereby PS performance automatically is scored, 
interpreted, and matched to instructional recommendations. 



PROJECT ACTIVITIES 



In implementing Project MAPS, the following series of actions were accomplished: 

We recruited and secured unidentified staff. 

Project staff attended weekly staff meetings, at which project activities, timelines, and 
accomplishments were discussed. 

We held quarterly focus group meetings where we reviewed MAPS plans and received input from 
key stakeholders about plans and activities. 

We identified teacher and student field testers and obtained informed consent. 

We conducted teacher workshops to prepare teachers for field-testing activities. Teachers field- 
tested software and provided interative feedback. 

We scripted and filmed video presentations of the PS situations and developed procedures for 
merging the video with multimedia presentation of the problem. We pressed the videos into CDs. 
We developed a platform by which to collect student responses so that automatic computerized 
scoring could occur. 

We developed a scoring system, by which every aspect of performance on every performance 
assessment received credit. We assessed the reliability of the scoring system and its validity against 
the global scoring rubric, which provided the basis for the scoring system. We programmed the 
computer to automatically complete the scoring. 

We field tested each component of the multimedia system separately and in combination. 

We conducted a large-scale study in which 40 students with disabilities and 40 comparable, same- 
age students completed performance assessments with and without the MAPS multimedia system 
in a cross-over, counterbalanced design. Results showed how the MAPS system could provide 
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support for students to show what they know on complex performance assessments. Also, results 
showed that students could be trained in the use of the software in large group sessions lasting no 
more than 45 minutes. 

The multimedia MAPS systems does the following: 

MAPS software was developed for 6 alternate performance assessments at each grade: 
grades 2, 3, 4, 5, and 6, for a total of 30 multimedia performance assessments. 

The multimedia presentation includes CD, options for reviewing selected parts of the CD, 
options for selecting parts of the text view (on screen), and options for selecting parts of the 
text for the computer to read/reread aloud. 

Students represent their responses on the computer by moving items (words, numbers, 
operation signs, etc.) from a sticker gallery or by typing in answers. 

The interactive scoring scoring system automatically scores the student’s responses, ft relies 
on elaborate frameworks for awarding credit for each aspect of the response for each 
question on each PA. Scores are provided for each assessment in terms of conceptual 
underpinning, computational applications, problem-solving strategies, and communicative 
value. In addition, the computer tracks progress on finer-level skills, which the teacher can 
use to inform her instructional planning. 

For student feedback, MAPS uses a video-game format, whereby the student accesses, at 
any time, the current score total. The computer also provides “tips,” which direct the student 
about how he can improve his response and his score. 

The teacher report summarizes salient dimensions of student performance at one point in 
time and over the course of time, in graphic and descriptive format. The report distinguishes 
between data representing first-time assessments (real assessment scores) and repeat 



administrations (i.e., practice) PA scores. 

See Appendix for sample screens for the MAPS system. 

CONCLUSIONS 

1 . A multimedia system can be designed to overcome barriers that (a) typically prevent teachers 
from having PSP As that feasibly and accurately index PS capacity among SWD and (b) 
facilitate PS instructional planning for SWD. 

2. Whereas available PSP As rely on text-based presentation and response formats, which 
spuriously lower estimates of PS capacity for SWD (who have reading and writing deficits), 
MAPS avoids reliance on these input and output modes. 

3. Although PSP As typically are time consuming for teachers to score and interpret, limiting their 
usefulness in schools, MAPS provides automatic scoring. 

4. Whereas teachers typically have difficulty connecting PSPA information with productive 
teaching methods for improving PS performance, MAPS structures teacher feedback in a 
manner that helps teachers connect complex assessment with instruction. 

5. Although students typically have to wait long time periods to obtain feedback on their 
performance and despite that feedback often is not useful to students for improving their 
performance, MAPS provides immediate and instructive feedback directly to students so that 



they may identify strategies for improving their own performance. 



6. Finally, most PSP As do not provide parallel forms for tracking PS improvement over time or 
for providing SWD with routine opportunities for practice. MAPS provides teachers with 
parallel forms of PAs within each grade at grades 2-6. 
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APPENDICES 



See attached sample screens from the MAPS multimedia system. 
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